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The findings from this review do not reflect the full body of research evidence on charter schools. 


What is this study about? 

The study measures the effects of attending Boston’s 
charter high schools on students’ reading and math 
achievement, high school graduation, and college out- 
comes. Six Boston charter schools that include one or 
more high school grades participated in the study. 

Researchers analyzed cohorts of students who applied 
to the study schools between 2002-09, a period when 
there were more applicants than the study schools 
could admit. When the number of applicants to a 
school exceeded enrollment limits, a lottery was used 
to randomly select students who would be offered 
admission. Students who were not selected were not 
offered admission to a charter school. Some students 
who were offered admission did not ultimately enroll 
in a charter school, and some students who were not 
initially offered admission through the lottery ultimately 
enrolled in a charter school. 

Study authors used statistical techniques to esti- 
mate the differences in outcomes for students who 
would enroll in a charter school if offered admission 
but would not enroll if they were assigned to the 
comparison group. These students are referred to as 
compliers. These types of estimates are known as a 
complier average causal effect (CACE). 3 

The study measured the CACE of attending a char- 
ter school on student achievement in reading and 
math, eligibility for a scholarship that waives Massa- 
chusetts public university tuition, high school gradu- 
ation, college enrollment, and college persistence. 


Features of Boston Charter Schools 


Charter schools are public schools that have been 
established on the basis of a contract, or charter, 
held by a private board of directors. They are 
exempt from many state and district regulations that 
govern traditional public schools, including those 
involving staffing, curriculum, and budget decisions. 
Charter schools are open to any student, and 
attendance is free of charge. If more students want 
to enroll in a charter school than the charter school 
can enroll, an admissions lottery must be held. 

Massachusetts’ urban charter schools typically 
use a No Excuses pedagogy, which emphasizes 
discipline, reading and math skills, extended 
instruction time, and selective teacher hiring . 4 Like 
most Boston charter schools, the Boston charter 
high schools included in this study largely utilize the 
No Excuses approach. 
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What did the study find? 



None of the analyses presented in this study meet 
WWC standards, and therefore, the study findings 
are not presented in this WWC report. 


The research described in this 
report does not meet WWC 
group design standards 


The study design is based on randomized offers 
of admission to charter schools. Using specialized 
statistical techniques, the study authors estimated 
the effects of enrollment in a charter school among 
students who would enroll in a charter school if 
offered admission and would not enroll in a charter 
school if they were not offered admission. These 
types of students are referred to as compliers, and 
these types of estimates are known as complier 
average causal effects (CACE). 

This study has high attrition from the initial 
randomization to the analytic sample, so the WWC 
requires a demonstration of baseline equivalence 
in order for this study to meet WWC group design 
standards. However, CACE analyses cannot 
demonstrate baseline equivalence of compliers 
because compliers cannot be identified . 5 Because 
the study cannot demonstrate baseline equivalence, 
the study does not meet WWC group design 
standards. Therefore, the findings from this study 
are not presented in this WWC report. 


March 2015 


Page 2 




WWC Single Study Review 


Endnotes 

1 Angrist, J. D., Cohodes, S. R., Dynarski, S. M., Pathak, P. A., & Walters, C. R. (2014). Stand and deliver: Effects of Boston’s charter 
high schools on college preparation, entry, and choice. Cambridge, MA: Department of Economics Working Paper, Massachusetts 
Institute of Technology. The authors also provided an updated version of the paper that has been accepted for publication in the 
Journal of Labor Economics, 201 5. This review used sample sizes from the updated version of the paper. An earlier version was used 
to complete a quick review of this study in July 2013: Angrist, J. A., Cohodes, S. R., Dynarski, S. M., Pathak, P. A., & Walters, C. D. 

(201 3). Charter schools and the road to college readiness: The effects on college preparation, attendance and choice. Boston, MA: The 
Boston Foundation and NewSchools Venture Fund. 

2 Single study reviews examine evidence published in a study (supplemented, if necessary, by information obtained directly from the 
authors) to assess whether the study design meets WWC design standards. The review reports the WWC’s assessment of whether 
the study meets WWC design standards and summarizes the study findings following WWC conventions for reporting evidence on 
effectiveness. This study was reviewed using the single study review protocol (version 2.0). A quick review of this study was released 
in July 2013, and this report is the follow-up review that replaces that initial assessment. 

3 The CACE is the average effect of taking up the intervention among compliers— those who would take up the intervention if assigned 
to the intervention group and who would not take up the intervention if assigned to the comparison group. In some disciplines, the 
CACE is also referred to as the local average treatment effect (LATE). The estimate of the offer of the intervention is often referred to as 
the intent-to-treat (ITT) effect. In studies that present both a CACE estimate and an ITT estimate, the WWC will review both estimates. 
The authors did not analyze ITT estimates, and therefore, only the CACE estimates are reviewed in this report. More information on the 
WWC guidance to review CACE estimates is available here: http://ies.ed. gov/ncee/wwc/documentsum.aspx?sid=253. 

4 Thernstrom, A. M., & Thernstrom, S. (2003). No excuses: Closing the racial gap in learning. New York: Simon & Schuster. 

5 Compliers are defined based on what their behavior would be if they were assigned to the intervention group and what their behavior 
would be if they were assigned to the comparison group. Because students can only be assigned to one research condition, student 
behavior under the other research condition is not known. Specifically, the compliers in the intervention condition cannot be distin- 
guished from individuals who would always take up the intervention (known as always-takers), and similarly, the compliers in the 
comparison condition cannot be distinguished from individuals who would never take up the intervention (known as never-takers). 

As a result, it is not possible to demonstrate the equivalence of compliers across condition, and therefore, any CACE estimate from a 
high-attrition randomized controlled trial cannot meet WWC group design standards. Only CACE estimates from low-attrition random- 
ized controlled trials can meet WWC group design standards without reservations. 
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Glossary of Terms 

Attrition 


Clustering adjustment 
Confounding factor 

Design 
Domain 
Effect size 

Eligibility 

Equivalence 

Improvement index 


Multiple comparison 
adjustment 

Quasi-experimental 
design (QED) 

Randomized controlled 
trial (RCT) 

Single-case design 
(SCD) 

Standard deviation 


Statistical significance 
Substantively important 


Attrition occurs when an outcome variable is not available for all participants initially assigned 
to the intervention and comparison groups. The WWC considers the total attrition rate and 
the difference in attrition rates across groups within a study. 

If intervention assignment is made at a cluster level and the analysis is conducted at the student 
level, the WWC will adjust the statistical significance to account for this mismatch, if necessary. 

A confounding factor is a component of a study that is completely aligned with one of the 
study conditions, making it impossible to separate how much of the observed effect was 
due to the intervention and how much was due to the factor. 

The design of a study is the method by which intervention and comparison groups were assigned. 
A domain is a group of closely related outcomes. 

The effect size is a measure of the magnitude of an effect. The WWC uses a standardized 
measure to facilitate comparisons across studies and outcomes. 

A study is eligible for review if it falls within the scope of the review protocol and uses either 
an experimental or matched comparison group design. 

A demonstration that the analytic sample groups are similar on observed characteristics 
defined in the review area protocol. 

Along a percentile distribution of individuals, the improvement index represents the gain 
or loss of the average individual due to the intervention. As the average individual starts at 
the 50th percentile, the measure ranges from -50 to +50. 

When a study includes multiple outcomes or comparison groups, the WWC will adjust 
the statistical significance to account for the multiple comparisons, if necessary. 

A quasi-experimental design (QED) is a research design in which study participants are 
assigned to intervention and comparison groups through a process that is not random. 

A randomized controlled trial (RCT) is an experiment in which eligible study participants are 
randomly assigned to intervention and comparison groups. 

A research approach in which an outcome variable is measured repeatedly within and 
across different conditions that are defined by the presence or absence of an intervention. 

The standard deviation of a measure shows how much variation exists across observations 
in the sample. A low standard deviation indicates that the observations in the sample tend 
to be very close to the mean; a high standard deviation indicates that the observations in 
the sample are spread out over a large range of values. 

Statistical significance is the probability that the difference between groups is a result of 
chance rather than a real difference between the groups. The WWC labels a finding statistically 
significant if the likelihood that the difference is due to chance is less than 5% (p < .05). 

A substantively important finding is one that has an effect size of 0.25 or greater, regardless 
of statistical significance. 


Please see the WWC Procedures and Standards Handbook (version 3.0) for additional details. 
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